Routine data in a primary care performance dashboard, Ethiopia

Abstract Objective To explore the feasibility of building a primary care performance dashboard using DHIS2 data from Ethiopia’s largest urban (Addis Ababa), agrarian (Oromia) and pastoral (Somali) regions. Methods We extracted 26 data elements reported by 12 062 health facilities to DHIS2 for the period 1 July 2022 to 30 June 2023. Focusing on indicators of effectiveness, safety and user experience, we built 14 indicators of primary care performance covering reproductive, maternal and child health, human immunodeficiency virus, tuberculosis, noncommunicable disease care and antibiotic prescription. We assessed data completeness by calculating the proportion of facilities reporting each month, and examined the presence of extreme outliers and assessed external validity. Findings At the regional level, average completeness across all data elements was highest in Addis Ababa (82.9%), followed by Oromia (66.2%) and Somali (52.6%). Private clinics across regions had low completeness, ranging from 38.6% in Somali to 58.7% in Addis Ababa. We found only a few outliers (334 of 816 578 observations) and noted that external validity was high for 11 of 14 indicators of primary care performance. However, the 12-month antiretroviral treatment retention rate and proportions of patients with controlled diabetes or hypertension exhibited poor external validity. Conclusion The Ethiopian DHIS2 contains information for measuring primary care performance, using simple analytical methods, at national and regional levels and by facility type. Despite remaining data quality issues, the health management information system is an important data source for generating health system performance assessment measures on a national scale.


Introduction
Functioning primary care is essential for improving population health.A greater emphasis on primary care in the health-care system can reduce costs, improve access and reduce inequities in population health.][6] Evidence on health system performance in Ethiopia has been limited by the availability and type of data used for assessment.Most studies have relied on data collected through household or health facility surveys, methods which are costly and infrequent. 7,8he government-owned health system is structured into three tiers: primary facilities such as health centres and, in rural areas, health posts and primary hospitals; secondary facilities (general hospitals); and tertiary facilities (specialized hospitals).The private health sector is present at all levels of care, ranging from primary care facilities to private for-profit hospitals and specialty centres. 9Outpatient primary care is delivered across all levels of the health system but it remains unclear whether primary care performance varies according to the facility type where it is provided.In particular, the performance of the Ethiopian private sector has received only limited attention in literature.
The capacity to measure and use data for learning and improvement is a foundation of a high-quality health system. 2 DHIS2 (formerly known as the District Health Information System 2) is an open-source, web-based health management information system platform designed to routinely generate health statistics to support decision-making.DHIS2 is a widely used platform, over 80 countries have adopted it, and DHIS2 supports routine health management for an estimated 3.2 billion people. 10Motivated by a need to improve data use for health system improvement, the Ethiopian health ministry adopted DHIS2 in 2016 as part of its digital health plan, achieving national implementation in 2019.The use of DHIS2 offers various benefits, including global support, standardization and flexibility.The system's open-source nature allows customization to meet each country's specific needs.The Ethiopian DHIS2 contains regularly submitted data, typically monthly, by all 30 192 health facilities in the country.The system includes reports on health service activities and the conditions for which people seek care. 11oncerns about data quality have hampered use of DHIS2.As a result, its rapid and extensive scaling up has not been matched by a corresponding increase in data use by policy-makers and researchers. 12DHIS2 data remain underrepresented in the scientific literature. 13The Ethiopian health ministry conducts annual reviews of health system performance using DHIS2 data. 14However, the majority of indicators monitored focus on coverage estimates (e.g.antenatal care coverage or immunization coverage) that do not reflect the quality of the services provided and rely on denominators that are sometimes inaccurate (e.g.estimated number of pregnant women, expected number of infants).
Beyond coverage measures, it remains unclear whether DHIS2 data can be used to track health system performance and to compare performance by facility type.In this paper, we explore the feasibility of building a primary care performance dashboard using DHIS2 data in three regions of Ethiopia.We conduct a detailed assessment of DHIS2 data quality by region and facility type, and discuss the availability of indicators to assess primary care performance.

Methods
Our analysis includes all health facilities reporting at least one of the indicators of interest to DHIS2 in three regions of Ethiopia: the largest urban (Addis Ababa), agrarian (Oromia) and pastoral (Somali) regions in the country.We extracted the monthly data at the health facility level.We categorized the facilities as: health posts (Oromia and Somali); health centres, private clinics, public hospitals and private hospitals (Addis Ababa and Oromia).Ethiopia uses its own 13-month calendar (the Ge'ez calendar).However, for budgeting and planning purposes the government uses a 12-month fiscal year that corresponds to 1 July to 30 June of the Gregorian calendar.For this analysis, we extracted data for Ethiopian Fiscal Year 2015, which corresponds to 1 July 2022 to 30 June 2023, using the pivot table module in DHIS2.We exported the data as csv files for analysis using Stata version 18 (Stata Corp. LP, College Station, United States of America).
To select sentinel measures of primary care performance, we first reviewed all data elements included in the Ethiopian DHIS2.Following the health system performance assessment framework for universal health coverage, we selected indicators of primary care effectiveness, safety and user experience. 15e aimed to include indicators that covered a range of primary care needs for different health conditions, including reproductive, maternal and child health indicators, human immunodeficiency virus (HIV), tuberculosis and noncommunicable disease care.We included measures of timely care (e.g.proportion of pregnant women who had their first antenatal care visit in the first trimester); appropriate care (e.g.proportion of pregnant women tested for syphilis and HIV during antenatal care); treatment effectiveness (tuberculosis treatment success rate, viral load suppression in patients living with HIV, blood pressure and blood sugar control); and retention in care (proportion of women attending four antenatal care visits of those having at least one visit; proportion of children receiving the third dose of pentavalent vaccine of those receiving the first dose; or proportion of children receiving the second dose of rotavirus vaccine of those receiving the first dose; and proportion of patients living with HIV still on antiretroviral therapy [ART] 12 months after initiation).The proportion of patients receiving an antibiotic was also included as a measure of treatment effectiveness, but also relates to patient safety and antimicrobial resistance. 16are retention reflects the user's experience and their willingness to continue receiving care.
We assessed four dimensions of data quality: reporting completeness, presence of outliers, internal consistency and external validity.For each data element, we assessed reporting completeness over the year by calculating the proportion of facilities reporting each month relative to the total number of facilities reporting at least once during the year.We also checked each data element for the presence of extreme positive outliers.We defined outliers as any observation greater than three standard deviations from the facility-level mean over the year, among volumes that were greater than 100 clients. 17We set any outlier found to be missing before describing the results.The statistical code used for the data quality assessment and removal of outliers is publicly available in an online repository. 18e assessed internal consistency by building the performance indicators (i.e.dividing one data element by the other) and ensuring that numerators did not surpass denominators at the regional and facility-type levels.0][21][22] The primary care performance dashboard was built using the data elements aggregated annually for the Ethiopian fiscal year 2015.The primary care performance indicators were also disaggregated by region and facility type.
The Institutional Review Board of The George Washington University determined that this study is not human subjects research, and exempted the study from a full review.

Results
From DHIS2, we extracted a total of 26 data elements, which we used to calculate 14 primary care performance indicators (Box 1).A total of 15 578 facilities were listed in DHIS2 across the three regions.However, 3516 did not report any of the 26 data elements needed for analysis during the year.Our analytical data set therefore included 12 062 health facilities and 144 744 facility-month observations.
Reporting completeness for each data element and region is shown in Table 1 (available at: https:// www .who.int/publications/ journals/ bulletin/ ).At the regional level, average completeness across all data elements was highest in Addis Ababa (82.9%), followed by Oromia (66.2%) and Somali (52.6%).Completeness was above 70% for all data elements in Addis Ababa except for tuberculosis.Previously, tuberculosis data elements were reported in Ethiopia on a quarterly basis.Some facilities may therefore be lagging in transitioning their reporting practices to a monthly frequency.In Oromia and Somali, seven and 11 data elements had completeness less than 50%, respectively.Facility deliveries and childhood vaccination had the highest reporting completeness (more than 75% in all three regions).Diabetes, hypertension and tuberculosis data elements had the lowest reporting completeness.Overall, we found few extreme outliers, less than 0.1% of observations (334/816 578).
Table 2 presents the primary care performance dashboard at the regional level as well as internal and external validity assessments.Only one indicator had poor internal consistency at the regional level, where the number of women receiving iron and folic acid during pregnancy in Oromia was slightly higher than the total number of first antenatal care visits, suggesting that iron and folic acid are delivered outside of antenatal care visits.Eleven out of 14 The proportions for diabetes and hypertension control also had poor external validity and were substantially higher than expected, at 77.6% and 77.4% on average, respectively (Table 2) compared with 34.4% and 37.5%, according to a systematic review and the 2023 WHO Global Report on Hypertension. 22,25HIS2 reporting for hypertension and diabetes is fairly recent in Ethiopia, and the numbers of patients enrolled in care may be poorly captured.
When assessing the proportion of total outpatient visits reported by each facility type, 15.8% of all visits across all three regions were reported by hospitals.Private sector primary care was low (less than 3.8%) in both Oromia and Somali, but accounted for 14.4% of reported primary care in Addis Ababa (Fig. 1).We assessed reporting completeness and internal consistency by region and facility type (Fig. 2).Completeness across all data elements was highest in Addis Ababa health centres (92.9% on average), followed by public hospitals in Addis Ababa and Oromia.In contrast, completeness was low in private clinics in all regions, ranging from 38.6% in Somali to 58.7% in Addis Ababa (Fig. 2 and online repository). 18The primary care performance indicators showed interesting trends by facility type (Table 3).For example, syphilis and HIV testing during antenatal care was not higher in hospitals compared with health centres, despite the former being generally better equipped.The 2021-2022 Service Provision Assessment survey also found that antenatal syphilis testing was equivalent in health centres (76%) compared with hospitals (75%). 8Furthermore, in Addis Ababa the ART retention rate was high (67.0%) in health centres, where reporting accuracy is notably high, compared with reported rates in other facility types (about 40.0%;Table 3).In the disaggregated analyses, two indicators had poor internal consistency.First, public hospitals in Addis Ababa saw twice as many women in their fourth or subsequent antenatal care visit compared with women in their first antenatal care visit (Table 3).This outcome is probably because some women were referred to public hospitals from the primary level for follow-up antenatal care.The primary care performance indicators by facility type may be biased if patients move to different levels of the health system throughout the year.Second, in health posts in Oromia, provision of iron and folic acid was six times the number of antenatal care visits, indicating that iron and folic acid are delivered outside of these visits

Discussion
In this paper, we have explored the feasibility of building a primary care performance dashboard using DHIS2 data from 12 062 health facilities in Ethiopia, corresponding to 40.0% of the 30 192 health facilities in the country.A total of 26 data elements were used to build 14 primary care performance indicators, including indicators of timely care, appropriate treatment, treatment effectiveness, safety and patient retention, which are important dimensions of service quality and intermediate objectives of health systems. 15ur data quality assessment revealed that reporting completeness is low in the Somali region compared with Addis Ababa and Oromia.Somali is one of four regions in Ethiopia where the predominant occupation is pastoralism.Furthermore, this region also has one of the weakest infrastructures in the country, characterized by a scarcity of health facilities and a shortage of health-care providers.Reporting completeness was also low in private facilities.Therefore, our estimate of the proportion of primary care provided by the private sector may be underestimated.The performance indicators in private facilities may also be biased downward if numerators had poorer completeness than denominators.Although we included all facilities reporting to DHIS2 over the year, some active facilities, particularly in the private sector, may still be missing from DHIS2.A Master Facility Registry, listing all existing facilities in Ethiopia, is being developed but is not currently integrated with DHIS2. 14Therefore, we are unsure of the true number of private facilities operating in the country.Poor reporting in the private sector has been described in other countries, and will require targeted approaches to incentivize private facilities to improve their reporting practices. 28,29Nonetheless, our findings on reporting completeness must be interpreted with caution.In DHIS2, low completeness may indicate true missing data, where facilities are failing to report, but can also mean that the facilities did not have any patients for a certain service in a particular month.Zero counts are not reported in DHIS2 and appear as missing in the data set, an important limitation of the platform that has been raised by others. 30ome facilities may also aggregate data over several months and only report once (e.g. per quarter) if the internet is not available in a given month, for example.Since we aggregated results over 12 months in our dashboard, only true missing data, where facilities are failing to report despite having patients, would bias the results.
Our findings also reveal that although most primary care services were delivered in public health centres or health posts, between 13.2% (7 622 941/57 718 718) and 28.3% (3 014 186/10 644 940) of total outpatient visits took place in hospitals.The provision of primary care in hospitals is not always recommended due to gaps in continuity, poorer user experience and higher costs. 31][33][34][35] DHIS2 data offer important advantages and opportunities for improving the assessment of health system performance in low-and middle-income countries.First, unlike population-or facility-based surveys conducted only every 4-5 years, DHIS2 data are reported monthly by all health facilities in the country, allowing frequent assessments on a national scale. 19For example, during the COVID-19 pandemic, many researchers turned to DHIS2 data to generate timely evidence on the magnitude of disruptions to health services. 4,5,10,12,36,37The use of one health management information system across all facility types and regions also facilitates standardized comparisons on a national scale.Moreover, unlike surveys, conducting performance assessments through DHIS2 does not require extra investments in data collection beyond the existing maintenance costs.Finally, DHIS2 data are locally led and government owned, decreasing reliance on international bodies for health system performance assessment.
Nonetheless, DHIS2 data continue to face limitations, including poor reporting completeness for some indicators, facilities or regions.This drawback will require improvements in data quality and reporting at the point of data collection.Upon reviewing all available data elements, we also observed certain ambiguities in indicator definitions.This issue has been documented in other countries as well. 30areful documentation of definitions and guidelines for reporting is crucial to improve data quality.To standardize reporting, WHO has collaborated with the health information systems programme at the University of Oslo, Norway, responsible for DHIS2, to create toolkits for specific programme areas (e.g.HIV and immunization) that include DHIS2 configuration packages. 38WHO should also include health system performance measures in these toolkits.In Ethiopia, DHIS2 data are also currently limited to facility-level aggregates.Incorporating patient-level information through

Oromia Somali
Health We have shown that the DHIS2 system in Ethiopia contains important measures of primary care performance and that, despite some data limitations, 11 of the indicators presented had good external validity.Previously, DHIS2 has been primarily used to estimate health intervention coverage, such as proportion of deliveries conducted in facilities or the proportion of children vaccinated.These estimates were often limited by unreliable denominators estimating

Fig. 1 .
Fig. 1.Primary care service volume by facility type in three regions of Ethiopia, July 2022 to June 2023

Data elements extracted from DHIS2 and primary care performance indicators calculated, Ethiopia, July 2022 to June 2023 Data elements (
8,27 Box 1. i) total no. of immediate postpartum contraceptive acceptors; (ii) total no. of births attended by skilled health personnel; (iii) no. of women attended first antenatal care visit; (iv) no. of first antenatal care visits in the first trimester;(v) no. of pregnant women attended four antenatal care visits; (vi) no. of pregnant women tested for syphilis; (vii) no. of pregnant women tested for HIV; (viii) no. of pregnant women who received iron and folic acid; (ix) no. of children with first dose of pentavalent vaccine; (x) no. of children with third dose of pentavalent vaccine; (xi) no. of children with first dose of rotavirus; (xii) no. of children with second dose of rotavirus; (xiii) no. of people still on ART 12 months after initiation; (xiv) no. of people initiated on ART; (xv) no. of ART patients with an undetectable viral load (< 50 copies/mL); (xvi) no. of ART patients for whom a viral load test was done at 12 months; (xvii) no. of tuberculosis patients cured; (xviii) total no. of tuberculosis patients on treatment; (xix) no. of hypertensive patients with controlled blood pressure at 6 months; (xx) no. of hypertensive patients enrolled in care 6 months prior; (xxi) no. of diabetic patients with controlled blood sugar at 6 months; (xxii) no. of diabetic patients enrolled in care 6 months prior; (xxiii) no. of patient encounters with one or more antibiotics; (xxiv) total no. of patient encounters at facilities; (xxv) total outpatient visits; and (xxvi) total number of new and repeat acceptor of oral contraceptives.
Antenatal care: proportion of pregnant women receiving timely antenatal care (timely care); proportion attended four antenatal care visits (care continuity); syphilis testing coverage; HIV testing coverage; and iron and folic acid provision (appropriate care) Routine immunization: retention to the third pentavalent vaccine dose; and retention to the second rotavirus vaccine dose (care continuity) HIV: proportion of people living with HIV on ART after 12 months (care continuity); and % of people with a viral load suppression (treatment effectiveness) Tuberculosis: treatment success rate (treatment effectiveness) Hypertension: proportion of patients with controlled blood pressure (treatment effectiveness) Diabetes: proportion of patients with controlled blood sugar (treatment effectiveness) Antibiotic prescribing: Proportion of patients receiving antibiotics (appropriate care, safety) Bull World Health Organ 2024;102:465-475A| doi: http://dx.doi.org/10.2471/BLT.23.291122Catherine Arsenault et al.Research

Table 2 . A primary care performance dashboard in three regions of Ethiopia, July 2022 to June 2023
ART: antiretroviral therapy; HIV: human immunodeficiency virus; NA: not applicable.a Number of facilities reporting varies for each data element.b Number of prescriptions for iron and folic acid was higher than number of antenatal care visits in Oromia region.c Indicators had poor external validity and results are likely inaccurate.Notes: We obtained the data from the Ethiopian DHIS2.Inconsistencies arise in some values due to rounding.Bull World Health Organ 2024;102:465-475A| doi: http://dx.doi.org/10.2471/BLT.23.291122Catherine Arsenault et al.Primary health care, Ethiopia

Table 3 . Primary care performance indicators by region and facility type, Ethiopia July 2022 to June 2023 Data element or indicator Addis Ababa
: Data elements definitions are provided in Box 1. Number of facilities per type and region is given in Table3.Data elements marked as NA are not reported by health posts.We obtained the data from the Ethiopian DHIS2.Completeness is defined as the number of facilities reporting each month divided by the total number of facilities that reported each data element at least once during the year.The table shows average completeness over 12 months.: Indicator definitions are provided in Box 1.We obtained the data from the Ethiopian DHIS2.Indicators marked as NA are not available in health posts.